Dynamic QoS Upgrading

ABSTRACT

In an embodiment, a memory controller includes multiple ports. Each port may be dedicated to a different type of traffic. In an embodiment, quality of service (QoS) parameters may be defined for the traffic types, and different traffic types may have different QoS parameter definitions. The memory controller may be configured to schedule operations received on the different ports based on the QoS parameters. In an embodiment, the memory controller may support upgrade of the QoS parameters when subsequent operations are received that have higher QoS parameters, via sideband request, and/or via aging of operations. In an embodiment, the memory controller is configured to reduce emphasis on QoS parameters and increase emphasis on memory bandwidth optimization as operations flow through the memory controller pipeline.

This application is a continuation of U.S. patent application Ser. No.12/883,878, filed Sep. 16, 2010. The above application is incorporatedherein by reference in its entirety.

BACKGROUND

1. Field of the Invention

This invention is related to the field of memory controllers.

2. Description of the Related Art

Digital systems generally include a memory system formed fromsemiconductor memory devices such as static random access memory (SRAM),dynamic random access memory (DRAM), synchronous DRAM (SDRAM), doubledata rate (DDR, DDR2, DDR3, etc.) SDRAM including low power versions(LPDDR, LPDDR2, etc.) SDRAM, etc. The memory system is volatile,retaining data when powered on but not when powered off, but alsoprovides low latency access as compared to nonvolatile memories such asFlash memory, magnetic storage devices such as disk drives, or opticalstorage devices such a compact disk (CD), digital video disk (DVD), andBluRay drives.

The memory devices forming the memory system have a low level interfaceto read and write the memory according to memory device-specificprotocols. The sources that generate memory operations typicallycommunicate via a higher level interface such as a bus, a point-to-pointpacket interface, etc. The sources can be processors, peripheral devicessuch as input/output (I/O) devices, audio and video devices, etc.Generally, the memory operations include read memory operations totransfer data from the memory to the device and write memory operationsto transfer data from the source to the memory. Read memory operationsmay be more succinctly referred to herein as read operations or reads,and similarly write memory operations may be more succinctly referred toherein as write operations or writes.

Accordingly, a memory controller is typically included to receive thememory operations from the higher level interface and to control thememory devices to perform the received operations. The memory controllergenerally also includes queues to capture the memory operations, and caninclude circuitry to improve performance. For example, some memorycontrollers schedule read memory operations ahead of earlier writememory operations that affect different addresses.

Memory controllers have limited visibility to the different types oftraffic that can be issued by the sources. Accordingly, memorycontrollers have not been able to segregate traffic at finer levels ofgranularity. Thus, performance improvements in memory controllers havebeen limited to the coarser mechanisms such as scheduling readoperations prior to write operations.

SUMMARY

In one embodiment, a memory controller is coupled to receive memoryoperations from multiple sources. Each source may assign a quality ofservice (QoS) parameter to each memory operation transmitted by thatsource. The memory controller may schedule the memory operations toaccess memory, and the QoS parameters assigned to the memory operationsmay be included in the scheduling determination. Additionally, in someembodiments, the memory controller may support upgrade of the QoSparameters of pending memory operations when a subsequent memoryoperation from the same flow is received with a QoS parameter specifyinga higher QoS level than the pending memory operations. The QoSparameters of the pending memory operation may be made equal to the QoSparameter of the newly-received memory operation. In this fashion, thememory controller may more quickly react to the increased needs of theflow. That is, the increase in QoS parameter for the memory operationmay indicate that the flow is not receiving data quickly, and thusincreasing the QoS of the pending memory operations to the QoS of thenewly-received operation may increase the rate at which the pendingoperations are serviced.

In some embodiments, memory operations from the same source but from adifferent flow may be pushed by a higher QoS operation. The pushedmemory operations may be treated as if they have the higher QoSparameter of the higher QoS operation. Pushing memory operations may beused, e.g., if the interface to the source completes memory operationsin the order issued by the sources. The lower-priority QoS operationsmay thus need to completed more quickly to permit the higher-priorityQoS operations to complete to the source.

BRIEF DESCRIPTION OF THE DRAWINGS

The following detailed description makes reference to the accompanyingdrawings, which are now briefly described.

FIG. 1 is a block diagram of one embodiment of a system including amemory controller.

FIG. 2 is a block diagram of one embodiment of QoS levels that may bedefined for RT and NRT classes of traffic.

FIG. 3 is a block diagram of one embodiment of the memory controllershown in FIG. 1.

FIG. 4 is a block diagram of one embodiment of a port interface unitthat may be included in one embodiment of an agent interface unit shownin FIG. 3.

FIG. 5 is a block diagram of one embodiment of a memory channelinterface unit that may be included in one embodiment of an agentinterface unit shown in FIG. 3.

FIG. 6 is a flowchart illustrating operation of one embodiment of a QoSarbiter shown in FIG. 5.

FIG. 7 is a block diagram of one embodiment of a pre-sorting queue (PSQ)illustrated in a memory channel unit in FIG. 3.

FIG. 8 is a flowchart illustrating operation of one embodiment of anenqueue control unit shown in FIG. 7.

FIG. 9 is a flowchart illustrating operation of one embodiment of ascheduler shown in FIG. 7 to select between reads and writes to bescheduled.

FIG. 10 is a flowchart illustrating operation of one embodiment of thescheduler shown in FIG. 7 to select a write operation to be scheduled.

FIG. 11 is a table illustrating operation of one embodiment of thescheduler shown in FIG. 7 to select a read to be scheduled.

FIG. 12 is a block diagram of one embodiment of a memory interface (MIF)unit shown in FIG. 3.

FIG. 13 is a flowchart illustrating operation of one embodiment of anMIF enqueue control unit shown in FIG. 12.

FIG. 14 is a flowchart illustrating operation of one embodiment of anactivate scheduler shown in FIG. 12.

FIG. 15 is a flowchart illustrating operation of one embodiment of acolumn address strobe (CAS) scheduler shown in FIG. 12.

FIG. 16 is a table illustrating operation of one embodiment of a finalscheduler shown in FIG. 12.

FIG. 17 is a block diagram of one embodiment of queuing at a high leveland relative importance of QoS parameters and memory bandwidthoptimization in scheduling at the various levels.

FIG. 18 is a block diagram of one embodiment of memory controllerfeatures to support in-band QoS upgrade.

FIG. 19 is flowchart illustrating operation of one embodiment of in-bandQoS upgrade.

FIG. 20 is a block diagram of one embodiment of a sideband QoS upgradeinterface.

FIG. 21 is a flowchart illustrating one embodiment of QoS upgrade due toaging of operations.

FIG. 22 is a block diagram of one embodiment of a system including anintegrated circuit illustrated in FIG. 1.

While the invention is susceptible to various modifications andalternative forms, specific embodiments thereof are shown by way ofexample in the drawings and will herein be described in detail. Itshould be understood, however, that the drawings and detaileddescription thereto are not intended to limit the invention to theparticular form disclosed, but on the contrary, the intention is tocover all modifications, equivalents and alternatives falling within thespirit and scope of the present invention as defined by the appendedclaims. The headings used herein are for organizational purposes onlyand are not meant to be used to limit the scope of the description. Asused throughout this application, the word “may” is used in a permissivesense (i.e., meaning having the potential to), rather than the mandatorysense (i.e., meaning must). Similarly, the words “include”, “including”,and “includes” mean including, but not limited to.

Various units, circuits, or other components may be described as“configured to” perform a task or tasks. In such contexts, “configuredto” is a broad recitation of structure generally meaning “havingcircuitry that” performs the task or tasks during operation. As such,the unit/circuit/component can be configured to perform the task evenwhen the unit/circuit/component is not currently on. In general, thecircuitry that forms the structure corresponding to “configured to” mayinclude hardware circuits. Similarly, various units/circuits/componentsmay be described as performing a task or tasks, for convenience in thedescription. Such descriptions should be interpreted as including thephrase “configured to.” Reciting a unit/circuit/component that isconfigured to perform one or more tasks is expressly intended not toinvoke 35 U.S.C. §112, paragraph six interpretation for thatunit/circuit/component.

DETAILED DESCRIPTION OF EMBODIMENTS

Turning now to FIG. 1, a block diagram of one embodiment of a system 5is shown. In the embodiment of FIG. 1, the system 5 includes anintegrated circuit (IC) 10 coupled to external memories 12A-12B. In theillustrated embodiment, the integrated circuit 10 includes a centralprocessor unit (CPU) block 14 which includes one or more processors 16and a level 2 (L2) cache 18. Other embodiments may not include L2 cache18 and/or may include additional levels of cache. Additionally,embodiments that include more than two processors 16 and that includeonly one processor 16 are contemplated. The integrated circuit 10further includes a set of one or more non-real time (NRT) peripherals 20and a set of one or more real time (RT) peripherals 22. In theillustrated embodiment, the RT peripherals include an image processor24, one or more display pipes 26, and a port arbiter 28. Otherembodiments may include more or fewer image processors 24, more or fewerdisplay pipes 26, and/or any additional real time peripherals asdesired. The image processor 24 may be coupled to receive image datafrom one or more cameras in the system 5. Similarly, the display pipes26 may be coupled to one or more display controllers (not shown) whichcontrol one or more displays in the system. In the illustratedembodiment, the CPU block 14 is coupled to a bridge/direct memory access(DMA) controller 30, which may be coupled to one or more peripheraldevices 32 and/or one or more peripheral interface controllers 34. Thenumber of peripheral devices 32 and peripheral interface controllers 34may vary from zero to any desired number in various embodiments. Thesystem 5 illustrated in FIG. 1 further includes a graphics unit 36comprising one or more graphics controllers such as G0 38A and G1 38B.The number of graphics controllers per graphics unit and the number ofgraphics units may vary in other embodiments. As illustrated in FIG. 1,the system 5 includes a memory controller 40 coupled to one or morememory physical interface circuits (PHYs) 42A-42B. The memory PHYs42A-42B are configured to communicate on pins of the integrated circuit10 to the memories 12A-12B. The memory controller 40 also includes a setof ports 44A-44E. The ports 44A-44B are coupled to the graphicscontrollers 38A-38B, respectively via a first type of interface (I/face1), respectively. The CPU block 14 is coupled to the port 44C via asecond type of interface (I/face 2). The NRT peripherals 20 and the RTperipherals 22 are coupled to the ports 44D-44E, respectively, via thesecond type of interface. The number of ports included in a memorycontroller 40 may be varied in other embodiments, as may the number ofmemory controllers. The number of memory PHYs 42A-42B and correspondingmemories 12A-12B may be one or more than two in other embodiments.

In one embodiment, each port 44A-44E may be associated with a particulartype of traffic. For example, in one embodiment, the traffic types mayinclude RT traffic, NRT traffic, and graphics traffic. Other embodimentsmay include other traffic types in addition to, instead of, or inaddition to a subset of the above traffic types. Each type of trafficmay be characterized differently (e.g. in terms of requirements andbehavior), and the memory controller may handle the traffic typesdifferently to provide higher performance based on the characteristics.For example, RT traffic requires servicing of each memory operationwithin a specific amount of time. If the latency of the operationexceeds the specific amount of time, erroneous operation may occur inthe RT peripheral. For example, image data may be lost in the imageprocessor 24 or the displayed image on the displays to which the displaypipes 26 are coupled may visually distort. RT traffic may becharacterized as isochronous, for example. On the other hand, graphicstraffic may be relatively high bandwidth, but is not latency-sensitive.NRT traffic, such as from the processors 16, is more latency-sensitivefor performance reasons but survives higher latency. That is, NRTtraffic may generally be serviced at any latency without causingerroneous operation in the devices generating the NRT traffic.Similarly, the less latency-sensitive but higher bandwidth graphicstraffic may be generally serviced at any latency. Other NRT traffic mayinclude audio traffic, which is relatively low bandwidth and generallymay be serviced with reasonable latency. Most peripheral traffic mayalso be NRT (e.g. traffic to storage devices such as magnetic, optical,or solid state storage).

By providing ports 44A-44E associated with different traffic types, thememory controller 40 may be exposed to the different traffic types inparallel, and may thus be capable of making better decisions about whichmemory operations to service prior to others based on traffic type. Insome embodiments, each port may be dedicated exclusively to either RT orNRT traffic. In other embodiments, the ports may be associated with theparticular traffic type, but other traffic types may be supported on theport. In an embodiment, graphics traffic may be considered to be NRTtraffic, but having visibility to the graphics traffic separate fromother NRT traffic may be useful for bandwidth balancing among the otherNRT sources and the RT source. Similarly, having the processor trafficseparate from the other NRT sources may be useful in bandwidthbalancing. In the illustrated embodiment, the RT port 44E may beassociated with RT traffic and the remaining ports 44A-44D may beassociated with NRT traffic.

Generally, a port may be a communication point on the memory controller40 to communicate with one or more sources. In some cases, the port maybe dedicated to a source (e.g. the ports 44A-44B may be dedicated to thegraphics controllers 38A-38B, respectively). In other cases, the portmay be shared among multiple sources (e.g. the processors 16 may sharethe CPU port 44C, the NRT peripherals 20 may share the NRT port 44D, andthe RT peripherals such as the display pipes 26 and the image processor24 may share the RT port 44E. A port may be coupled to a singleinterface to communicate with the one or more sources. Thus, whensources share an interface, there may be an arbiter on the sources' sideof the interface to select between the sources. For example, the L2cache 18 may serve as an arbiter for the CPU port 44C to the memorycontroller 40. The port arbiter 28 may serve as an arbiter for the RTport 44E, and a similar port arbiter (not shown) may be an arbiter forthe NRT port 44D. The single source on a port or the combination ofsources on a port may be referred to as an agent.

Each port 44A-44E is coupled to an interface to communicate with itsrespective agent. The interface may be any type of communication medium(e.g. a bus, a point-to-point interconnect, etc.) and may implement anyprotocol. In some embodiments, the ports 44A-44E may all implement thesame interface and protocol. In other embodiments, different ports mayimplement different interfaces and/or protocols. For example, theembodiment illustrated in FIG. 1 includes the graphics controllers38A-38B using one type of interface/protocol and the CPU block 14, theNRT peripherals 20 and the RT peripherals 22 using another type ofinterface/protocol. An interface may refer to the signal definitions andelectrical properties of the interface, and the protocol may be thelogical definition of communications on the interface (e.g. includingcommands, ordering rules, coherence support if any, etc.). Supportingmore than one interface on the various ports 44A-44E may eliminate theneed to convert from one interface supported by a source/agent and theinterface supported by the memory controller 40, which may improveperformance in some embodiments. Additional, some embodiments of theintegrated circuit 10 may include circuitry purchased from a third partyas a prepackaged unit typically referred to in the industry asintellectual property (IP). The IP may be “hard” (in which case it isspecified as a netlist of circuits that are laid out and placed on theIC as a block) or “soft” (in which case it is specified as asynthesizable block that can be synthesized with other blocks to beincluded in the integrated circuit 10). Both soft and hard IP include aspecified interface and protocol, which generally cannot be changed bythe designer of the integrated circuit 10 (at least without payment offees to the third party that owns the IP). Accordingly, supportingmultiple interfaces/protocols may permit easier inclusion of third partyIP.

In an embodiment, each source may assign a quality of service (QoS)parameter to each memory operation transmitted by that source. The QoSparameter may identify a requested level of service for the memoryoperation. Memory operations with QoS parameter values requesting higherlevels of service may be given preference over memory operationsrequesting lower levels of service. For example, each of the interfacesto the ports 44A-44E in FIG. 1 are shown to include a command (Cmd), aflow identifier (FID), and a QoS parameter (QoS). The command mayidentify the memory operation (e.g. read or write). A readcommand/memory operation causes a transfer of data from the memory12A-12B to the source, whereas a write command/memory operation causes atransfer of data from the source to the memory 12A-12B. Commands mayalso include commands to program the memory controller 40. For example,which address ranges are mapped to which memory channels, bandwidthsharing parameters, etc. may all be programmable in the memorycontroller 40. The FID may identify a memory operation as being part ofa flow of memory operations. A flow of memory operations may generallybe related, whereas memory operations from different flows, even if fromthe same source, may not be related. A portion of the FID (e.g. a sourcefield) may identify the source, and the remainder of the FID mayidentify the flow (e.g. a flow field). Thus, an FID may be similar to atransaction ID, and some sources may simply transmit a transaction ID asan FID. In such a case, the source field of the transaction ID may bethe source field of the FID and the sequence number (that identifies thetransaction among transactions from the same source) of the transactionID may be the flow field of the FID. Sources that group transactions asa flow, however, may use the FIDs differently. Alternatively, flows maybe correlated to the source field (e.g. operations from the same sourcemay be part of the same flow and operations from a different source arepart of a different flow). The ability to identify transactions of aflow may be used in a variety of ways described below (e.g. QoSupgrading, reordering, etc.).

Thus, a given source may be configured to use QoS parameters to identifywhich memory operations are more important to the source (and thusshould be serviced prior to other memory operations from the samesource), especially for sources that support out-of-order datatransmissions with respect to the address transmissions from the source.Furthermore, the QoS parameters may permit sources to request higherlevels of service than other sources on the same port and/or sources onother ports.

In some embodiments, different traffic types may have differentdefinitions of QoS parameters. That is, the different traffic types mayhave different sets of QoS parameters. The meaning of a given QoSparameter value depends on the set of QoS parameters from which it isdrawn. For example, a set of RT QoS parameters may be defined and a setof NRT QoS parameters may be defined. Thus, an RT QoS parameter value isassigned meaning within the RT QoS parameter set and an NRT QoSparameter value is assigned meaning within the NRT QoS parameter set.Other embodiments may implement the same QoS parameter set on all portsor among all traffic types.

The memory controller 40 may be configured to process the QoS parametersreceived on each port 44A-44E and may use the relative QoS parametervalues to schedule memory operations received on the ports with respectto other memory operations from that port and with respect to othermemory operations received on other ports. More specifically, the memorycontroller 40 may be configured to compare QoS parameters that are drawnfrom different sets of QoS parameters (e.g. RT QoS parameters and NRTQoS parameters) and may be configured to make scheduling decisions basedon the QoS parameters.

The QoS parameters may thus be values that are transmitted with memoryoperations, and which may be used in the memory controller to identifyrequested QoS levels. The QoS levels may be relative to other levels,and may specify which memory operations are preferred to be selectedprior to others having lower QoS levels. Thus, the QoS levels mayfunction as a sort of priority after interpretation by the memorycontroller 40 to consider QoS levels defined in different sets, althoughthe priority can be balanced by other factors.

In some embodiments, the memory controller 40 may be configured toupgrade QoS levels for pending memory operations. Various upgrademechanism may be supported. For example, the memory controller 40 may beconfigured to upgrade the QoS level for pending memory operations of aflow responsive to receiving another memory operation from the same flowthat has a QoS parameter specifying a higher QoS level. This form of QoSupgrade may be referred to as in-band upgrade, since the QoS parameterstransmitted using the normal memory operation transmission method alsoserve as an implicit upgrade request for memory operations in the sameflow. The memory controller 40 may be configured to push pending memoryoperations from the same port or source, but not the same flow, as anewly received memory operation specifying a higher QoS level. Asanother example, the memory controller 40 may be configured to couple toa sideband interface from one or more agents, and may upgrade QoS levelsresponsive to receiving an upgrade request on the sideband interface. Inanother example, the memory controller 40 may be configured to track therelative age of the pending memory operations. The memory controller 40may be configured to upgrade the QoS level of aged memory operations atcertain ages. The ages at which upgrade occurs may depend on the currentQoS parameter of the aged memory operation.

The memory controller 40 may be configured to determine the memorychannel addressed by each memory operation received on the ports, andmay be configured to transmit the memory operations to the memory12A-12B on the corresponding channel. The number of channels and themapping of addresses to channels may vary in various embodiments and maybe programmable in the memory controller. The memory controller may usethe QoS parameters of the memory operations mapped to the same channelto determine an order of memory operations transmitted into the channel.That is, the memory controller may reorder the memory operations fromtheir original order of receipt on the ports. Additionally, duringprocessing in the channel, the memory operations may be reordered againat one or more points. At each level of reordering, the amount ofemphasis placed on the QoS parameters may decrease and factors thataffect memory bandwidth efficiency may increase. Once the memoryoperations reach the end of the memory channel pipeline, the operationsmay have been ordered by a combination of QoS levels and memorybandwidth efficiency. High performance may be realized in someembodiments.

The processors 16 may implement any instruction set architecture, andmay be configured to execute instructions defined in that instructionset architecture. The processors 16 may employ any microarchitecture,including scalar, superscalar, pipelined, superpipelined, out of order,in order, speculative, non-speculative, etc., or combinations thereof.The processors 16 may include circuitry, and optionally may implementmicrocoding techniques. The processors 16 may include one or more level1 caches, and thus the cache 18 is an L2 cache. Other embodiments mayinclude multiple levels of caches in the processors 16, and the cache 18may be the next level down in the hierarchy. The cache 18 may employ anysize and any configuration (set associative, direct mapped, etc.).

The graphics controllers 38A-38B may be any graphics processingcircuitry. Generally, the graphics controllers 38A-38B may be configuredto render objects to be displayed into a frame buffer. The graphicscontrollers 38A-38B may include graphics processors that may executegraphics software to perform a part or all of the graphics operation,and/or hardware acceleration of certain graphics operations. The amountof hardware acceleration and software implementation may vary fromembodiment to embodiment.

The NRT peripherals 20 may include any non-real time peripherals that,for performance and/or bandwidth reasons, are provided independentaccess to the memory 12A-12B. That is, access by the NRT peripherals 20is independent of the CPU block 14, and may proceed in parallel with CPUblock memory operations. Other peripherals such as the peripheral 32and/or peripherals coupled to a peripheral interface controlled by theperipheral interface controller 34 may also be non-real timeperipherals, but may not require independent access to memory. Variousembodiments of the NRT peripherals 20 may include video encoders anddecoders, scaler/rotator circuitry, image compression/decompressioncircuitry, etc.

As mentioned above, the RT peripherals 22 may include the imageprocessor 24 and the display pipes 26. The display pipes 26 may includecircuitry to fetch one or more frames and to blend the frames to createa display image. The display pipes 26 may further include one or morevideo pipelines. The result of the display pipes 26 may be a stream ofpixels to be displayed on the display screen. The pixel values may betransmitted to a display controller for display on the display screen.The image processor 24 may receive camera data and process the data toan image to be stored in memory.

The bridge/DMA controller 30 may comprise circuitry to bridge theperipheral(s) 32 and the peripheral interface controller(s) 34 to thememory space. In the illustrated embodiment, the bridge/DMA controller30 may bridge the memory operations from the peripherals/peripheralinterface controllers through the CPU block 14 to the memory controller40. The CPU block 14 may also maintain coherence between the bridgedmemory operations and memory operations from the processors 16/L2 Cache18. The L2 cache 18 may also arbitrate the bridged memory operationswith memory operations from the processors 16 to be transmitted on theCPU interface to the CPU port 44C. The bridge/DMA controller 30 may alsoprovide DMA operation on behalf of the peripherals 32 and the peripheralinterface controllers 34 to transfer blocks of data to and from memory.More particularly, the DMA controller may be configured to performtransfers to and from the memory 12A-12B through the memory controller40 on behalf of the peripherals 32 and the peripheral interfacecontrollers 34. The DMA controller may be programmable by the processors16 to perform the DMA operations. For example, the DMA controller may beprogrammable via descriptors. The descriptors may be data structuresstored in the memory 12A-12B that describe DMA transfers (e.g. sourceand destination addresses, size, etc.). Alternatively, the DMAcontroller may be programmable via registers in the DMA controller (notshown).

The peripherals 32 may include any desired input/output devices or otherhardware devices that are included on the integrated circuit 10. Forexample, the peripherals 32 may include networking peripherals such asone or more networking media access controllers (MAC) such as anEthernet MAC or a wireless fidelity (WiFi) controller. An audio unitincluding various audio processing devices may be included in theperipherals 32. One or more digital signal processors may be included inthe peripherals 32. The peripherals 32 may include any other desiredfunctional such as timers, an on-chip secrets memory, an encryptionengine, etc., or any combination thereof.

The peripheral interface controllers 34 may include any controllers forany type of peripheral interface. For example, the peripheral interfacecontrollers may include various interface controllers such as auniversal serial bus (USB) controller, a peripheral componentinterconnect express (PCIe) controller, a flash memory interface,general purpose input/output (I/O) pins, etc.

The memories 12A-12B may be any type of memory, such as dynamic randomaccess memory (DRAM), synchronous DRAM (SDRAM), double data rate (DDR,DDR2, DDR3, etc.) SDRAM (including mobile versions of the SDRAMs such asmDDR3, etc., and/or low power versions of the SDRAMs such as LPDDR2,etc.), RAIVIBUS DRAM (RDRAM), static RAM (SRAM), etc. One or more memorydevices may be coupled onto a circuit board to form memory modules suchas single inline memory modules (SIMMs), dual inline memory modules(DIMM5), etc. Alternatively, the devices may be mounted with theintegrated circuit 10 in a chip-on-chip configuration, apackage-on-package configuration, or a multi-chip module configuration.

The memory PHYs 42A-42B may handle the low-level physical interface tothe memory 12A-12B. For example, the memory PHYs 42A-42B may beresponsible for the timing of the signals, for proper clocking tosynchronous DRAM memory, etc. In one embodiment, the memory PHYs 42A-42Bmay be configured to lock to a clock supplied within the integratedcircuit 10 and may be configured to generate a clock used by the memory12.

It is noted that other embodiments may include other combinations ofcomponents, including subsets or supersets of the components shown inFIG. 1 and/or other components. While one instance of a given componentmay be shown in FIG. 1, other embodiments may include one or moreinstances of the given component. Similarly, throughout this detaileddescription, one or more instances of a given component may be includedeven if only one is shown, and/or embodiments that include only oneinstance may be used even if multiple instances are shown.

It is noted that other embodiments of the memory controller 40 may notimplement multiple ports, but may still implement QoS parameters,different QoS parameters/levels for different traffic types or classessuch as NRT and RT, and QoS upgrading.

Turning next to FIG. 2, a pair of tables 50 and 52 are shownillustrating a definition of a set of RT QoS levels and a set of NRT QoSlevels, respectively, for one embodiment. Other embodiments may includeadditional or substitute levels, and other embodiments may includeadditional levels in combination with a subset of the illustratedlevels. As illustrated by the arrows pointing downward next to thetables 50 and 52 in FIG. 2, the tables illustrate the QoS levels withina set in increasing priority. That is, the real time green (RTG) QoSlevel is the lowest priority RT QoS level; the real time yellow (RTY)QoS level is the medium priority RT QoS level; and the real time red(RTR) QoS level is the highest priority RT QoS level. Similarly, thebest effort (BEF) QoS level is the lowest priority NRT QoS level and thelow latency (LLT) QoS level is the highest priority NRT QoS level. Theillustration of the RT QoS levels and the NRT QoS levels next to eachother in FIG. 2 is not intended to indicate the relative priority of theRT QoS levels with regard to the NRT QoS levels. Instead, the memorycontroller 40 may determine such relative priorities based, in part, onother factors indicating the traffic that is being experienced by thememory controller 40 across the various types and ports.

The RTG, RTY, and RTR QoS levels may reflect relative levels of urgencyfrom an RT source. That is, as the amount of time before data is neededby the RT source to prevent erroneous operation decreases, the QoS levelassigned to each memory operation increases to indicate the higherurgency. By treating operations having higher urgency with higherpriority, the memory controller 40 may return data to the RT source morequickly and may thus aid the correct operation of the RT source.

For example, the display pipe 26 may initiate the reading of frame datafrom the memory 12A-12B for the next frame to be displayed in thevertical blanking interval for the display. The frame is not actuallydisplayed until the end of the vertical blanking interval, and thus thedisplay pipe 26 may use the RTG level during this time period. As theframe begins to be displayed (i.e. the display controller begins readingframe pixels from the display pipe 26 output), the display pipe 26 mayraise the QoS level of frame data read operations to the memory to theRTY level. For example, if the amount of frame data that is read aheadof the current pixel being displayed reduces below a first threshold,the level may be raised to RTY. At a second threshold (lower than thefirst threshold), the display pipe 26 may raise the QoS level of memoryoperations to RTR.

The BEF NRT QoS level may be a request to return the data as quickly asthe memory controller 40 is able, once the needs of other flows of dataare met. On the other hand, the LLT NRT QoS level may be a request forlow latency data. NRT memory operations having the LLT QoS level may betreated more closely, in terms of priority with other memorytransactions, than those having the BEF QoS level (at least in somecases). In other cases, the BEF and LLT QoS levels may be treated thesame by the memory controller 40.

Turning next to FIG. 3, a block diagram of one embodiment of the memorycontroller 40 is shown. In the embodiment of FIG. 3, the memorycontroller 40 includes an agent interface unit (AIU) 54 and one or morememory channel units 56A-56B. There may be one memory channel unit56A-56B for each memory channel included in a given embodiment, andother embodiments may include one channel or more than two channels. Asillustrated in FIG. 3, the AIU 54 may include multiple port interfaceunits 58A-58E. More particularly, there may be a port interface unit58A-58E for each port 44A-44E on the memory controller 40. The AIU 54may further include memory channel interface units (MCIUs) 60A-60B (onefor each memory channel unit 56A-56B). The AIU 54 may further includeone or more bandwidth sharing registers 62, which may be programmable toindicate how bandwidth is to be shared among the ports. The portinterface units 58A-58E may be coupled to receive memory operations andto receive/transmit data and responses on the corresponding port, andmay also be coupled to the MCIUs 60A-60B. The MCIUs 60A-60B may furtherbe coupled to the bandwidth sharing registers 62 and to thecorresponding MCU 56A-56B. As illustrated in FIG. 3, the MCUs 56A-56Bmay each include a presorting queue (PSQ) 64 and a memory interfacecircuit (MIF) 66. The PSQs 64 are coupled to the corresponding MCIUs60A-60B and to the MIF 66 in the same MCU 56A-56B. The MIF 66 in eachMCU 56A-56B is coupled to the corresponding memory PHY 42A-42B.

The AIU 54 may be configured to receive memory operations on the ports44A-44E and to switch the memory operations to the channels addressed bythose memory operations, using the QoS parameters of the memoryoperations as a factor in deciding which memory operations to transmitto one of the MCUs 56A-56B prior to other memory operations to the sameMCU 56A-56B. Other factors may include the bandwidth sharing controls todivide bandwidth on the memory channels among the ports.

More particularly, each port interface unit 58A-58E may be configured toreceive the memory operations from the corresponding port 44A-44E, andmay be configured to determine the memory channel to which a givenmemory operation is directed. The port interface unit 58A-58E maytransmit the memory operation to the corresponding MCIU 60A-60B, and maytransmit reads separately from writes in the illustrated embodiment.Thus, for example, the port interface unit 58A may have a Rd0 connectionand a Wr0 connection to the MCIU 60A for read operations and writeoperations, respectively. Similarly, the port interface unit 58A mayhave a Rd1 and a Wr1 connection to the MCIU 60B. The other portinterface units 58B-58E may have similar connections to the MCIU60A-60B. There may also be a data interface to transmit read data fromthe port interface units 58A-58B to the MCIUs 60A-60B, illustratedgenerally as the dotted “D” interface for the MCIU 60A in FIG. 3.

The MCIUs 60A-60B may be configured to queue the memory operationsprovided by the port interface units 58A-58E, and to arbitrate among thememory operations to select operations to transmit to the correspondingMCUs 56A-56B. The arbitration among operations targeted at a givenmemory channel may be independent of the arbitration among operationstargeted at other memory channels.

The MCIUs 60A-60B may be coupled to the bandwidth sharing registers 62,which may be programmed to indicate how memory bandwidth on a channel isto be allocated to memory operations in the given channel. For example,in one embodiment, the MCIUs 60A-60B may use a deficit-weightedround-robin algorithm to select among the ports when the is no highpriority traffic present (e.g. RTR or RTY QoS levels in the RT traffic).When RTR or RTY traffic is present, a round-robin mechanism may be usedto select among the ports that have RTR/RTY traffic. The weights in thedeficit weighted round-robin mechanism may be programmable to allocatedrelatively more bandwidth to one port than another. The weights may beselected to favor processor traffic over the graphics and NRT ports, forexample, or to favor the graphics ports over other ports. Any set ofweights may be used in various embodiments. Other embodiments maymeasure the bandwidth allocations in other ways. For example,percentages of the total bandwidth may be used. In other embodiments, acredit system may be used to control the relative number of operationsfrom each port that are selected. Generally, however, operations may beselected based on both QoS parameters and on bandwidth sharingrequirements in various embodiments.

The MCUs 56A-56B are configured to schedule memory operations from theirqueues to be transmitted on the memory channel. The MCUs may beconfigured to queue reads and writes separately in the PSQs 64, and maybe configured to arbitrate between reads and writes using a credit basedsystem, for example. In the credit-based system, reads and writes areallocated a certain number of credits. The number of write credits andread credits need not be equal. Each scheduled memory operation mayconsume a credit. Once both the write credits and the read credits arereduced to zero or less and there is a pending transaction to bescheduled, both credit may be increased by the corresponding allocatednumber of credits. Other embodiments may use other mechanisms to selectbetween reads and writes. In one embodiment, the credit system may bepart of the arbitration mechanism between reads and writes (along withmeasurements of the fullness of the write queue). That is, as the writequeue becomes more full, the priority of the writes in the arbitrationmechanism may increase. Additional details are set forth below.

In one embodiment, the QoS parameters of the write operations may beeliminated on entry into the PSQs 64. The read operations may retain theQoS parameters, and the QoS parameters may affect the read schedulingfrom the PSQs 64.

In an embodiment, the MCUs 56A-56B may schedule memory operations inbursts of operations (each operation in the burst consuming a credit).If the burst reduces the credit count to zero, the burst may bepermitted to complete and may reduce the credit count to a negativenumber. When the credit counts are increased later, the negative creditsmay be accounted for, and thus the total number of credits afterincrease may be less than the allocated credit amount.

To create bursts of memory operations for scheduling, the MCUs 56A-56Bmay group memory operations into affinity groups. A memory operation maybe said to exhibit affinity with another memory operation (or may besaid to be affine to the other memory operation) if the operations maybe performed efficiently on the memory interface when performed in closeproximity in time. Efficiency may be measured in terms of increasedbandwidth utilization. For example, SDRAM memories are characterized bya page that can be opened using an activate command (along with anaddress of the page). The size of the page may vary from embodiment toembodiment, and generally may refer to a number of contiguous bits thatmay be available for access once the activate command has beentransmitted. Asynchronous DRAM memories may similarly have a page thatmay be opened by asserting a row address strobe control signal and byproviding the row address. Two or more memory operations that accessdata in the same page may be affine, because only one activate/RAS maybe needed on the interface for the memory operations. SDRAM memoriesalso have independent banks and ranks. A bank may be a collection ofmemory cells within an SDRAM chip that may have an open row (withinwhich page hits may be detected). A rank may be selected via a chipselect from the memory controller, and may include one or more SDRAMchips. Memory operations to different ranks or banks may also be affineoperations, because they do not conflict and thus do not require thepage to be closed and a new page to be opened. Memory operations may beviewed as affine operations only if they transfer data in the samedirection (i.e. read operations may only be affine to other readoperations, and similarly write operations may only be affine otherwrite operations). Memory operations to the same page (or to an openpage) may be referred to as page hits, and memory operations todifferent banks/ranks may be referred to as bank hits and rank hits,respectively.

The MCUs 56A-56B may also be configured to schedule commands on thememory interface to the memories 12A-12B (through the memory PHYs42A-42B) to perform the scheduled memory operations. More particularly,in an embodiment, the MCUs 56A-56B may be configured to presynthesizethe commands for each memory operation and to enqueue the commands. TheMCUs 56A-56B may be configured schedule the commands to provideefficient use of the memory bandwidth. The MIFs 66 in each MCU 56A-56Bmay implement the presynthesis of commands and the scheduling of thecommands, in an embodiment.

Turning now to FIG. 4, a block diagram of one embodiment of the portinterface unit 58C is shown. Other port interface circuits 58A-58B and58D-58E may be similar, although there may be differences inimplementation for port interface circuits that couple to differentinterfaces. In the illustrated embodiment, the port interface unit 58Cincludes buffers 70A-70B coupled to read (AR) and write (AW) interfacesto receive read and write memory operations, respectively, asillustrated in FIG. 4. The buffers 70A-70B are coupled to a read spawngenerator 72 and a write spawn generator 74, respectively, which arecoupled to the Rd0/Rd1 interfaces and the Wr0/Wr1 interfaces,respectively. The read spawn generator 72 is coupled to a readoutstanding transaction table (ROTT) 76, and the write spawn generator74 is coupled to a write outstanding transaction table (WOTT) 78. TheROTT 76 is coupled to a read response generator 80 which is configuredto generate a read response on the interface. The ROTT is also coupledto a read buffer 84, which is coupled to receive data from either MCU56A-56B through a mux 86 and to provide read data on the interface. TheWOTT 78 is coupled to a write response generator 82 which is configuredto generate a write response on the interface. The WOTT 78 is alsocoupled to a write data forward buffer 88, which is coupled to providedata to the MCUs 56A-56B and is coupled to receive data from a buffer70C, which is coupled to receive write data from the interface.

For a read operation, the buffer 70A may be configured to receive theoperation from the interface. The buffer 70A may be provided to capturethe read operation and hold it for processing by the read spawngenerator 72. In an embodiment, the buffer 70A may be a two entry “skid”buffer that permits a second operation to be captured in the event ofdelay for an unavailable resource to become available, for example, thuseasing timing on propagating back pressure requests to the source(s) onthe interface. The buffers 70B-70C may similarly be two entry skidbuffers. Other embodiments may include additional entries in the skidbuffers, as desired.

The read spawn generator 72 may be configured to decode the address ofthe read operation to determine which memory channel is addressed by theread operation. The read spawn generator 72 may be configured totransmit the read operation to the addressed memory channel via the Rd0or Rd1 interface. In some embodiments, a read operation may overlapmemory channels. Each read operation may specify a size (i.e. a numberof bytes to be read beginning at the address of the operation). If thecombination of the size and the address indicates that bytes are readfrom more than one channel, the read spawn generator 72 may beconfigured to generate multiple read operations to the addressedchannels. The read data from the multiple read operations may beaccumulated in the read buffer 84 to be returned to the source.

The read spawn generator 72 may also be configured to update the ROTT76, allocating an entry in the ROTT 76 to track the progress of theread. Once the data has been received in the read buffer 84, the ROTT 76may be configured to signal the read response generator 80 to generate aread response to transfer the data to the source. If read data is to bereturned in order on the interface (e.g. according to the protocol onthe interface), the data may remained buffered in the read buffer 84until previous reads have been returned and then the ROTT 76 may signalthe read response generator 80 to transfer the data. The ROTT 76 may becoupled to receive various status signals from the MCUs 56A-56B toupdate the status of the pending read operations (not shown in FIG. 4).

The buffer 70B, the write spawn generator 74, and the WOTT 78 mayoperate similarly for write operations. However, data is received ratherthan transmitted on the interface. The write data may be received in thewrite data forward buffer 88, and may be forwarded to the currentlocation of the corresponding write operation. The WOTT 78 may signalfor the write response once the write has been guaranteed to complete,terminating the writes on the interface with a write response earlierthan might otherwise be possible.

It is noted that, while the embodiment illustrated in FIG. 4 includes aninterface that conveys read and write memory operations separately (ARand AW, respectively), other embodiments may include a singletransmission medium for both read and write operations. In such anembodiment, a single buffer 70 may receive the operations, and the readspawn generator 72 and the write spawn generator 74 may decode thecommand from the interface to differentiate read and write operations.Alternatively, there may be one spawn generator which generates bothread and write operations and updates the ROTT 76 or the WOTT 78accordingly.

Turning now to FIG. 5, a block diagram illustrating one embodiment ofthe MCIU 60A is shown. The MCIU 60B may be similar except that it iscoupled to receive the Rd1 and Wr1 inputs from each port and is coupledto the MCU 56B. In the illustrated embodiment, the MCIU includes a setof read queues such as read queues 90A-90B shown in FIG. 5 and a set ofwrite queues such as write queues 92A-92B. There may be one read queueand one write queue for each port. Each read queue is coupled to the Rd0output of one of the port interface units 58A-58E and is coupled to aQoS arbiter 94A. Each write queue is coupled to the Wr0 outputs of arespective one of the port interface units 58A-58E and to a QoS arbiter94B. The outputs of the QoS arbiters 94A-94B are provided to the MCU 56Aas read and write inputs, respectively. The QoS arbiters 94A-94B arecoupled to receive data from the bandwidth sharing registers 62.

Two read queue entries are shown in the read queue 90A, and other readqueues may be similar. The read queue 90A includes the FID of the memoryoperation, the QoS parameter of the operation, a push bit (P), and another field (Oth). The FID and the QoS parameter may be the same valuesthat were transmitted with the memory operation on the interface to thememory controller 40. Alternatively, one or both values may be recodedby the memory controller 40 for convenience internally. The push bit maybe used to force a higher priority on the memory operation if a secondmemory operation is ordered behind the memory operation and is a higherQoS level then than the memory operation. For example, the second memoryoperation may be received on the same port as the memory operation, andthe interface on that port may require that data be returned in the sameorder that the memory operations are transmitted. By forcing the higherpriority, the memory operation may be performed more quickly and maythus permit a more rapid servicing of the second memory operation withthe higher QoS level. The other field may include various otherinformation for the memory operation (e.g. the address, sizeinformation, etc.). Similarly, two write queue entries are shown in thewrite queue 92A, and may include fields similar to the read queue 90A.The other field may store different information for writes as comparedto reads, if desired.

The QoS arbiters 94A-94B may arbitrate among the read queues 90A-90B andthe write queues 92A-92B, respectively. The QoS arbiters may take intoaccount both QoS levels (as indicated by the QoS parameters in thequeues) and the bandwidth sharing parameters from the bandwidth sharingregisters. The selected read and write operation (if any) is transmittedto the MCU 56A.

It is noted that, in some embodiments, there may be one or more bypasspaths from a port interface unit to the QoS arbiters 94A and/or 94B (notshown in FIG. 5). For example, reads from the CPU port interface unit58C may bypass to the QoS arbiter 94A to permit lower latency reads tothe processors. In some embodiments, one or more memory operations inthe queues 90A-90B and/or 92A-92B may be masked from the QoS arbiters94A-94B. For example, write operations for which the corresponding datahas not yet reached the memory controller 40 may be masked from the QoSarbiters 94A-94B, to avoid blocking write operations from other portsfor which corresponding data has reached the memory controller 40.

FIG. 6 is a flowchart illustrating operation of one embodiment of eachof the QoS arbiters 94A-94B. While blocks are shown in a particularorder for ease of understanding, other orders may be used. Blocks may beperformed in parallel in combinatorial logic in the QoS arbiters94A-94B. Blocks, combinations of blocks, and/or the flowchart as a wholemay be pipelined over multiple clock cycles. The QoS arbiters 94A-94Bmay be configured to implement the operation illustrated in FIG. 6.

If at least one memory operation in one of the queues to which the QoSarbiter is coupled has an RT QoS level of RTY or RTR, or the push bit isset for at least one memory operation (decision block 100, “yes” leg),the QoS arbiter may arbitrate among the queues having the RTY or RTR QoSlevels and/or set push bits (block 102). Since each queue corresponds toa port in this embodiment, the QoS arbiter may effectively bearbitrating among the ports having the RTY or RTR QoS levels and/or setpush bits. Thus, the AIU 54 may treat the RTG, BEF, and LLT QoS levelsas equal in this embodiment. The AIU 54 may treat the RTR and RTY QoSlevels and set push bit operations as equal to each other and higherpriority than the other levels in this embodiment. Other embodiments mayhave additional levels of granularity (e.g. RTR may be treated as higherpriority that RTY, RTG and LLT may be treated as higher priority thanBEF, etc.). Additionally, other embodiments may implement a differentscheme than round robin.

On the other hand, if there are no memory operations having the RTY orRTR QoS levels and no memory operations having set push bits (decisionblock 100, “no” leg), the QoS arbiter may arbitrate among all queues(all ports) (block 104). For example, the QoS arbiter may implement adeficit-weighted round robin scheme among the ports, where the weightsmay be based on the bandwidth sharing parameters (or the bandwidthsharing parameters may be the weights). Other embodiments may implementother arbitration schemes among all the ports.

Turning now to FIG. 7, a block diagram of one embodiment of the PSQ 64is shown. In the illustrated embodiment, the PSQ 64 includes an enqueuecontrol unit 110, a set of transaction queues 112, a scheduler 114, anda bypass mux 116. The enqueue control unit 110 is coupled to receive theread and write operations from the corresponding MCIU 60A or 60B, and iscoupled to the transaction queues 112. The transaction queues 112 arefurther coupled to the scheduler 114, the output of which is coupled tothe bypass mux 116. The bypass mux 116 is coupled to receive the readoperation as well, and is configured to select between the memoryoperation scheduled by the scheduler 114 and the read operation. Forexample, the read operation may bypass the transaction queues 112 ifthere are no reads in the transaction queues 112 and the number ofwrites is below a threshold level. Other embodiments may not implementbypassing and the bypass mux 116 may be eliminated.

As illustrated in FIG. 7, the transaction queues 112 may include a setof read affinity queues such as queues 118A-118B and a set of writeaffinity queues such as queues 120A-120B. The number of read affinityqueues and write affinity queues may vary from embodiment to embodiment,and the number of read affinity queues need not equal the number ofwrite affinity queues. Each affinity queue may store one or more memoryoperations that have been determined by the enqueue control unit 110 toexhibit affinity with each other. Thus, as a memory operation isreceived by the enqueue control unit 110, the enqueue control unit 110may be configured to compare the memory operation to the affinity queues118A-118B (for a read operation) or the affinity queues 120A-120B (for awrite operation). If the memory operation is affine, it may be enqueuedin the corresponding affinity queue. If not, the memory operation may beenqueued in another affinity queue. In an embodiment, a read affinityqueue may be reserved for reads that are not affine, and similarly awrite affinity queue may be reserved for writes that are not affine.

The scheduler 114 may be configured to schedule memory operations to betransmitted to the MIF 66. For read operations, the scheduler 114 may beconfigured to consider both QoS levels in the read affinity queues118A-118B and the number of affine memory operations in each readaffinity queue 118A-118B. More details of one embodiment are describedbelow. Generally, however, the scheduler 114 may be configured to favorread operations that have high QoS levels and larger numbers of affinememory operations. For write operations, the QoS levels may beeliminated in the PSQ 64. That is, the QoS levels of the writeoperations may be dropped as the write operations are written to thetransaction queues 112. As illustrated in the exemplary entries in eachof the queues 118A and 120A, the read operations may retain QoS whilethe write operations may not. The scheduler 114 may be configured toschedule between read operations and write operation based on fullnessof the write queues and QoS levels in the read queues, for example.Additional details will be provided further below.

It is noted that, in some embodiments, the read affinity queues118A-118B and write affinity queues 120A-120B may be physicallyinstantiated (e.g. as separate data structures or as one or more datastructures that are divided via logic circuitry in the PSQ 64 and/orprogrammably separated). In other embodiments, the affinity queues maybe virtual. That is, there may be a read queue and a write queue, forexample, and tagging may be used to identify affine operations.

FIG. 8 is a flowchart illustrating operation of one embodiment of theenqueue control unit 110 in response to receive a memory operation.While blocks are shown in a particular order for ease of understanding,other orders may be used. Blocks may be performed in parallel incombinatorial logic in the enqueue control unit 110. Blocks,combinations of blocks, and/or the flowchart as a whole may be pipelinedover multiple clock cycles. The enqueue control unit 110 may beconfigured to implement the operation illustrated in FIG. 8. The enqueuecontrol unit 110 may be configured to implement the operationillustrated in FIG. 8 in parallel for a concurrently-received readoperation and write operation.

The enqueue control unit 110 may compare the received operation to thecorresponding affinity queues 118A-118B or 120A-120B to determine if thereceived operation exhibits affinity with the queued operations(decision block 130). In response to detecting affinity (and if there isspace in the affinity queue for the operation), the enqueue control unit110 may enqueue the operation in the affinity queue (block 132). Inresponse to not detecting affinity, the enqueue control unit 110 mayenqueue the operation in an empty affinity queue (block 134).

The detection of affinity is shown in exploded view in FIG. 8, for oneembodiment. An operation may be affine if it is a page hit with otheroperations in the affinity queue (decision block 136, “yes” leg), or abank or rank miss with the other operations (decision blocks 138 and140, respectively, “yes” legs). If not (decision blocks 136, 138, and140, “no” legs), the operation is not affine with the queued operations.The detection illustrated by decision blocks 136, 138, and 140 may beperformed in parallel for each affinity queue.

FIG. 9 is a flowchart illustrating operation of one embodiment of thescheduler 114 for scheduling between read operations and writeoperations in the transaction queues 112. While blocks are shown in aparticular order for ease of understanding, other orders may be used.Blocks may be performed in parallel in combinatorial logic in thescheduler 114. Blocks, combinations of blocks, and/or the flowchart as awhole may be pipelined over multiple clock cycles. The scheduler 114 maybe configured to implement the operation illustrated in FIG. 9.

In the embodiment of FIG. 9, there are three thresholds for write queuefullness (that is, the number of write operations in the transactionqueues 112): high, medium, and low. The high level indicates more writeoperations in the transaction queues 112 than the medium levelindicates, and the medium level indicates more write operations in thetransaction queues 112 than the low level indicates. The thresholds maybe fixed or programmable, in various embodiments.

The scheduler 114 may be configured to detect a blocking write (decisionblock 150, “yes” leg), and may be configured to schedule the blockingwrite (block 152). A blocking write may be a write operation that hasthe push bit set (indicating that a memory operation having a higher QoSlevel is ordered behind the write operation). A blocking write operationmay also be a write operation that is blocking a read operation to thesame address (or the write operation updates at least one byte read bythe read operation).

If there are no blocking writes, the scheduling of read operation vs.write operations may be based on a credit system. That is, readoperations and write operations may each be assigned a certain number ofcredits. A scheduled memory operation may consume one credit from thecorresponding credit count. In the illustrated embodiment, once the readcredits are exhausted and a read operation is scheduled, the credits forboth reads and writes may reload by adding the initial credit values tothe current counts. Because affine operations may be scheduled in aburst when a read or write operation is scheduled, the credit counts maybe reduced below zero. That is, the scheduler 114 may not interrupt aburst of affine operations if the credit count for the operations isreduced to zero. For brevity, the discussion below with respect to FIG.9 may refer to read operations in the read queue and write operations inthe write queue. The read queue may include the combination of the readaffinity queues 118A-118B and the write queue may include thecombination of the write affinity queues 120A-120B.

Responsive to detecting that the write queue has reached the highthreshold and there is at least one write credit available (decisionblock 154, “yes” leg), the scheduler 114 may be configured to schedulefrom the write queue (block 156). If the write queue has not reached thehigh threshold or there are no available write credits (decision block154, “no” leg), but the scheduler 114 determines that there is at leastone read operation in the read queue and there is an available readcredit (decision block 158, “yes” leg), the scheduler 114 may beconfigured to schedule from the read queue (block 160). If there are noread operations in the read queue or there are no available read credits(decision block 158, “no” leg) but the scheduler 114 determines that thewrite queue has reached the medium threshold and there is an availablewrite credit (decision block 162, “yes” leg), the scheduler may beconfigured to schedule from the write queue (block 156). If the writequeue has not reached the medium threshold or there are no availablewrite credits (decision block 162, “no” leg), but the scheduler 114determines that there is at least one read operation in the read queueand there is no available read credit (decision block 164, “yes” leg),the scheduler 114 may be configured to schedule from the read queue(block 160). As mentioned previously, the scheduler 114 may beconfigured to reload read and write credits in this instance as well. Ifthere is no read operation in the read queue (decision block 164, “no”leg) and the scheduler 114 determines that the write queue has reachedthe low threshold (decision block 166, “yes” leg), the scheduler 114 maybe configured to schedule from the write queue (block 156).

While the flowchart of FIG. 9 illustrates not scheduling from the writequeue until the write queue reaches the low threshold, some embodimentsmay schedule from the write queue if the scheduler 114 has been idle fora period of time. The period of time may be fixed or programmable.

FIG. 10 is a block diagram illustrating operation of one embodiment ofthe scheduler 114 to implement scheduling from the write queue (block156). As illustrated in FIG. 10, the scheduler 114 may be configured toschedule the write affinity queue 120A-120B having the largest number ofwrite operations (block 168).

FIG. 11 is a table illustrating various states that the scheduler 114may be in to schedule read operations from the read queue (block 160).The states are illustrated in increasing order of priority. Thus, ifmore than one entry in the table of FIG. 11 would correspond to thecontents of the read queue, the entry with the highest priority may bethe state of the scheduler 114. The scheduler 114 may be configured toschedule operations as illustrated in the “schedule” column of FIG. 11for the corresponding state.

The table of FIG. 11 may refer to the number of reads in the read queuebeing above or below a green or yellow threshold. The green and yellowthresholds may be different from the RTG and RTY QoS levels, and may besimilar to write queue thresholds discussed above. The green and yellowthresholds may be fixed or programmable. Additionally, an affinity groupmay be referred to as being scheduled. An affinity group may be thegroup of operations in one of the read affinity queues 118A-118B.

The scheduler read state may be green if the number of reads in the readqueue is below the yellow threshold (and has not been above the yellowthreshold since entering the green state), and if there are no RTY orRTR reads in the queue. In the green state, the scheduler 114 may beconfigured to schedule an LLT read operation, if any. The operations inthe read affinity queue with the LLT read operation may also bescheduled, if any. If there are no LLT read operations, the affinitygroup having the oldest read operation may be scheduled.

The scheduler read state may be LLT yellow if the number of reads isabove the green threshold (or has not fallen below the green thresholdsince the LLT yellow state was entered) and there are no RTR or RTY readoperations in the read queue. In the LLT yellow state, the scheduler 114may be configured to schedule LLT read operations, if any. If there areno LLT read operations, the scheduler 114 may be configured to schedulethe largest affinity group in the read queue. In an embodiment, thescheduler 114 may be programmable to schedule the affinity groupcontaining the oldest read operation if there are no LLT read operationsin the LLT yellow state, similar to the green state.

The scheduler read state may be yellow if there is at least one RTY readoperation but no RTR operations. In the yellow state, the scheduler 114may be configured to scheduling the affinity group that includes theoldest RTY read operation, or the largest affinity group.

The scheduler read state may be red affinity if there is at least oneRTR read operation in the queue and the scheduler 114 is programmed toassociate affinity operations with RTR operations. In the red affinitystate, the scheduler 114 may be configured to schedule the affinitygroup that includes the oldest RTR operation. Otherwise, the largestaffinity group may be scheduled.

The scheduler read state may be red if the there is at least one RTRread operation in the queue and the scheduler 114 is programmed toschedule RTR read operations without their affine operations. In the redstate, the scheduler 114 may be configured to schedule the oldest RTRread operation, or the largest affinity group.

The above discussion has referred to reads in the read queue as being“oldest.” In some embodiments, read operations may be viewed as oldestbased on time spent in the read queue. In other embodiments, age may bemeasured in other ways. For example, the oldest read operation may bemeasured in terms of the number of times the read operation has not beenscheduled in favor of another read operation.

When an affinity group is scheduled, the number of memory operationsscheduled from the affinity group may be the number of operations in thegroup up to a fixed or programmable burst size. In an embodiment, if acurrently scheduled affinity group has fewer operations than the burstsize, other memory operations may be scheduled. The scheduler 114 may,for example, schedule “stragglers” (e.g. memory operations that werepart of an affinity group that was previously scheduled but terminateddue to burst size) that may be affine with the currently scheduledaffinity group. Alternatively, the scheduler 114 may be configured toselect the next largest affinity group.

Turning now to FIG. 12, a block diagram of one embodiment of the MIF 66is shown. In the embodiment of FIG. 12, the MIF 66 includes an MIFenqueue control unit 170, MIF transaction queues 172, an open page table174, a precharge scheduler 176, a mode register (reg) scheduler 178, arefresh scheduler 180, an activate scheduler 182, a column addressstrobe (CAS) scheduler 184, and a final scheduler 186. The MIF enqueuecontrol unit 170 is coupled to receive a memory operation from the PSQ64, and is coupled to the open page table 174 and the MIF transactionqueues 172. The open page table 174 is coupled to the prechargescheduler 176. The MIF transaction queues 172 include an activate queue188 and a CAS queue 190 which are coupled to the activate scheduler 182and the CAS scheduler 184, respectively. The schedulers 176, 178, 180,182, and 184 are coupled to the final scheduler 186, which is coupled totransmit a command to one of the memory PHYs 42A or 42B depending onwhether the MIF 66 is instantiated in the MCU 56A or 56B.

The MIF enqueue control unit 170 may be configured to compare thereceived memory operation to the open page table 174 and the MIFtransaction queues 172, and may be configured to presynthesize commandsto perform the memory operation. Particularly, if the page correspondingto the received memory operation will be opened by a queued activatecommand, the received memory operation may be queued as a CAS operation.If the page corresponding to the received memory operation is alreadyopen and there are no queued page-conflict activates, the receivedmemory operation may be queued as a CAS operation. If the pagecorresponding to the received memory operation is not open, the memoryoperation may be queued as an activate command and a CAS command. Eachcommand may be assigned an age counter (age field in the activate queue188 and the CAS queue 190). The value of the age counter may be based onthe QoS parameter of a read memory operation, and may be shorter forhigher QoS levels than for lower QoS levels. For write memoryoperations, the age counter may be the same as the lowest QoS level forread memory operations, or may be longer than the lowest QoS level. Agemay be counted as a number of operations which bypass the operation, oras a number of clock cycles, in various embodiments. Once an age counteris reduced to zero, the corresponding operation may be scheduled as thehighest priority in the MIF 66.

The activate scheduler 182 and the CAS scheduler 184 may be configuredto schedule activate commands and CAS commands from the correspondingqueues 188 and 190. The activate command may open a page in the memoryand the CAS command may cause a read or a write of data within the page.

The precharge scheduler 176 may be configured to schedule a prechargecommand for a bank when the idle timer for the page in the open pagetable 174 expires. The idle timer may be reloaded each time a receivedmemory operation hits in the open page, and may be decremented eachclock cycle or for each memory operation which does not hit in the openpage. In cases in which a page conflict with a newly received memoryoperation causes a page to be closed, the last CAS command to theprevious page may be used to precharge the bank.

The mode reg scheduler 178 may be configured to generate memory registerread (MRR) and memory register write (MRW) commands according toconfigurable timers and/or in response to memory timer requests. Therefresh scheduler 180 may generate refresh commands when refresh isneeded.

The final scheduler 186 may select among the schedulers 176, 178, 180,182, and 184 to schedule a command to the memory PHY circuit. Thescheduler 186 may also enforce certain protocols such as the turnaroundcycle between read and write operations, etc.

It is noted that the activate and CAS queues 188 and 190 may be virtual.That is, the same entry in the MIF transaction queues 172 may be usedfor both the activate and the CAS command for a given memory operation.If an activate is needed for the given memory operation, the entry maybe initialized as an activate command and may be converted to a CAScommand once the activate is issued. If an activate is not needed forthe given memory operation, the entry may be initialized as a CAScommand.

It is further noted that, in addition to the circuitry illustrated inFIGS. 7 and 12, there may be a data path (not shown) in the MCUs 56A-56Bto transport write data from the AIU 54 to the memory and to transportread data from the memory to the AIU 54.

FIG. 13 is a flowchart illustrating operation of one embodiment of theMIF enqueue control unit 170 in response to a memory operation from thePSQ 64. While blocks are shown in a particular order for ease ofunderstanding, other orders may be used. Blocks may be performed inparallel in combinatorial logic in the MIF enqueue control unit 170.Blocks, combinations of blocks, and/or the flowchart as a whole may bepipelined over multiple clock cycles. The MIF enqueue control unit 170may be configured to implement the operation illustrated in FIG. 13.

The MIF enqueue control unit 170 may be configured to determine if anactivate command is needed for the received memory operation (e.g. thememory operation does not hit an open page or a page that will be openedby a previously queued activate command) (decision block 196). If so(decision block 196, “yes” leg), the MIF enqueue control unit 170 may beconfigured to synthesize an activate command and enqueue the activatecommand in the activate queue 188 (block 192). The MIF enqueue controlunit 170 may also be configured to synthesize a CAS command and enqueuethe CAS command in the CAS queue 190 (block 194). In each case, the agecounter may be initialized (based on the QoS parameter for reads).

FIG. 14 is a flowchart illustrating operation of one embodiment of theactivate scheduler 182. The activate scheduler 182 may be configured toscan the activate commands in the activate queue 188 for an activatecommand that has aged out (e.g. it's age counter is zero). If such anactivate command is detected, the activate may select the aged outactivate command (block 200). Otherwise, the activate scheduler 182 mayselect the activate command in the head cluster that is in the bankhaving the most pending operations. The head cluster may be the group ofactivates of the same read/write type that are the oldest activatecommands in the activate queue 188. If more than one activate commandqualifies, the oldest qualifying activate command may be selected.

FIG. 15 is a flowchart illustrating operation of one embodiment of theCAS scheduler 184. While blocks are shown in a particular order for easeof understanding, other orders may be used. Blocks may be performed inparallel in combinatorial logic in the CAS scheduler 184. Blocks,combinations of blocks, and/or the flowchart as a whole may be pipelinedover multiple clock cycles. The CAS scheduler 184 may be configured toimplement the operation illustrated in FIG. 15.

Like the activate scheduler 182, the CAS scheduler 184 may be configuredto detect aged-out CAS commands and select those for transmission first(decision block 202, “yes” leg and block 204). If there are no aged-outCAS commands (decision block 202, “no” leg) and the most recent CAScommand was a read (decision block 206, “yes” leg), the CAS scheduler184 may be configured to select the next read CAS command to the samerank as the last CAS command or to select the read CAS command with themost conflicting operations waiting for the CAS command to complete sothat the page can be closed (block 208). Similarly, if the last CAScommand was a write (decision block 206, “no” leg and decision block210, “yes” leg), the CAS scheduler 184 may be configured to select thenext write CAS command to the same rank as the last CAS command or toselect the write CAS command with the most conflicting operationswaiting for the CAS command to complete so that the page can be closed(block 212). If none of the above apply, the CAS scheduler 184 may beconfigured to select the oldest CAS command in the CAS queue 190 (block214). It is noted that, in blocks 204, 208, 212, and 214, if more thanone CAS command qualifies, the oldest CAS command may be selected.

FIG. 16 is a table illustrating operation of one embodiment of the finalscheduler 186. The entries in the table are illustrated in decreasingpriority from top to bottom. If a burst stop (BST) command or aninterrupt CAS command is to be transmitted, these commands may be givenhighest priority. Next may be a command from the mode reg scheduler 178,followed by an auto-refresh command from the refresh scheduler 180. ACAS command from the CAS scheduler 184 may be next in priority to theauto-refresh command, then the activate command from the activatescheduler 182. A precharge command from the precharge scheduler 176 mayfollow the activate command in priority, and lowest in priority in FIG.16 may be the self-refresh command used to enter a low power mode.

As discussed briefly above, a memory operation may enter the memorycontroller 40 from a source with a high importance placed on the QoSlevel of the memory operation. As the memory operation is processed inthe memory controller 40 and approaches transmission to the memory, theimportance of the QoS level may decrease and the importance of memory(SDRAM) efficiency may increase. FIG. 17 illustrates this operationdiagrammatically. The MCIU 60 (e.g. 60A or 60B), the PSQ 64, and the MIF66 are illustrated, along with blocks illustrating the schedulingcontrols at each level. At the MCIU 60, the QoS levels of both read andwrite operations, along with the bandwidth sharing parameters, maycontrol the selection of read and write operations for transmission tothe PSQ 64. Memory efficiency may not be considered at this level (block222). At the PSQ 64, the QoS for writes is dropped. Affinity controlsthe selection of writes, and a combination of QoS and affinity mayprimarily control the selection of reads (block 224). Accordingly, QoSand memory efficiency concerns may be more balanced at this PSQ 64. Atthe MIF 66, the QoS is dropped for all memory operations (although QoSmay be reflected in aging parameters), and SDRAM efficiency concerns mayprimarily control the selection of commands (block 226).

Also as discussed briefly above, the memory controller 40 may supportthe upgrade of the QoS level of pending memory operations. Variousembodiments may support one or more mechanisms to signal an upgrade ofthe QoS level: in-band, side band, and aging.

FIG. 18 is a block diagram illustrating one embodiment of the in-bandQoS upgrade mechanism. In this mechanism, receiving a memory operationin the same flow as a previous memory operation, and the newly-receivedmemory operation having a higher QoS level than the previous memoryoperation, may cause the memory controller 40 to upgrade the QoS levelof the previous memory operation. A portion of the MCIU 60A isillustrated, along with a portion of the PSQ 64 in the MCU 56A. The MCIU60B and the PSQ 64 in the MCU 56B may be similar.

The MCIU 60A may receive memory operations from each port as previouslydiscussed, and may queue the newly received memory operations in one ofthe queues 90A-90B or 92A-92B for reads and writes, respectively.Additionally, as illustrated in FIG. 18, the MCIU 60A may compare theFID and QoS parameters of the newly received memory operation to thecorresponding values of the queued memory operations from the same port(e.g. the RdQ0 90A may compare the FID and QoS parameters of the newlyreceived memory operation on port 0, or G0 port 44A, and the RdQ4 90Bmay compare the FID and QoS parameters for the newly received memoryoperation on port 4, or the RT port 44E). In response to a newlyreceived memory operation from the same port and flow having a higherQoS level as a queued memory operation, the MCIU 60A may upgrade the QoSlevel of the queued memory operation to the higher QoS level. In someembodiments, QoS upgrade may be limited to certain ports. For example,one embodiment may limit QoS upgrade to the RT port 44E. Otherembodiments may limit the QoS upgrade to ports on which RT traffic maybe received. In such embodiments, only the newly received memoryoperations from the ports for which upgraded is supported may becompared.

Similarly, FID and QoS parameters may be received by the read affinityqueues 118A-118B in the PSQ 64. The PSQ 64 may upgrade the QoS level ofa queued read operation in response to a newly received memory operationfrom the same port and flow that has a higher QoS level. In someembodiments, the FID and QoS parameters received by the PSQ 64 may bethe same newly-received FID and QoS parameters as those received by theMCIU 60A. That is, the PSQ 64 and the MCIU 60A may upgrade QoS levels inparallel for each newly received memory operation. In other embodiments,the MCIU 60A may perform QoS upgrade in response to the newly receivedmemory operation, and the PSQ 64 may perform QoS upgrade in response toa memory operation being transmitted from the MCIU 60A to the PSQ 64.

While FIG. 18 illustrates comparing FID and QoS parameters to the readqueues 90A-90B, comparison may also be made to the write queues 92A-92Bto upgrade the QoS parameters. As discussed previously, however, the QoSparameters for write operations may be dropped in the PSQ 66, and thusthere may be no comparison to the write affinity queues 120A-120B insuch embodiments.

FIG. 19 is a flowchart illustrating operation of one embodiment of thememory controller 40 to upgrade QoS levels for the in-band upgrademechanism. While the blocks are shown in FIG. 19 in a particular orderfor ease of understanding, other orders may be used. Blocks,combinations of blocks, and/or the flowchart as a whole may be pipelinedover multiple clock cycles. The memory controller 40 may be configuredto implement the operation in FIG. 19. More particularly, each of thePSQ 64 and the MCIUs 60A-60B may include upgrade circuitry configured toimplement the operation shown in FIG. 19.

If a newly received memory operation is from the same port as a queuedmemory operation (decision block 230, “yes” leg), from the same flow asindicated by the FID (decision block 232, “yes” leg), and the queuedmemory operation is an RT memory operation (decision block 234, “yes”leg), the memory controller 40 may upgrade the QoS level of the queuedoperation to the higher QoS level of the newly received memory operation(block 236). In one embodiment, the queues in which QoS upgrade issupported may include content addressable memories to perform thecompare, and a match may enable a write of the newly-received QoSparameter to the matching queue entry.

If the newly received memory operation is from the same port and flow asa queued memory operation (decision blocks 230 and 232, “yes” legs), butthe queue memory operation is not an RT memory operation (decision block234, “no” leg), the memory controller 40 may be configured to set the Pbit for the queued memory operation (block 238). Similarly, if the newlyreceived memory operation is from the same port but a different flow asa queued memory operation (decision block 230, “yes” leg and decisionblock 232, “no” leg), the memory controller 40 may be configured to setthe P bit for the queued memory operation (block 238).

FIG. 20 is a block diagram illustrating the addition of a sidebandinterface to request a QoS upgrade. In the embodiment of FIG. 20, thesideband interface is included for the RT port 44E. Other embodimentsmay implement one or more additional sideband interfaces for otherports. As the name implies, the sideband interface may be in addition tothe “regular” interface for transmitting memory operations to the memorycontroller 40. The regular interface (including the Cmd, FID, and QoSparameter as shown in FIG. 1) is illustrated via arrow 240. The sidebandinterface may include a sideband QoS upgrade request signal (SbQoSUpgd),a sideband QoS parameter (SbQoS), a sideband FID (SbFID), a sidebandmask (SbMask), and a sideband read indication (SbRd) from the RTperipherals 22 to the RT port 44E, and a sideband acknowledge (SbAck)from the RT port 44E to the RT peripherals 22.

The RT peripherals 22 may assert the SbQoSUpgd signal to request asideband upgrade, and may supply the QoS parameter representing theupgraded QoS level on the SbQoS. The SbFID may identify the flow, andthe SbMask may provide for generality by permitting a portion of theSbFID and the queued FIDs to be masked in the comparison. The SbRdindicator may indicate if reads or writes are to be compared forupgrade. The RT port 44E may acknowledge the upgrade request using theSbAck. In one embodiment, the hardware within the memory controller 40to perform the sideband upgrade may be similar to that shown in FIG. 18,and the process maybe similar to that shown in FIG. 19. The newlyreceived memory operation may be replaced in the above description bythe sideband request being received.

FIG. 21 is a flowchart illustrating generalized operation of the memorycontroller 40 for implementing aging-based QoS upgrade. As mentionedpreviously, the MIF transaction queues 172 may implement priorityupgrade based on aging. Other embodiments may similarly implementaging-based QoS upgrades. While the blocks are shown in a particularorder in FIG. 21 for ease of understanding, other orders may be used.Blocks may be performed in parallel in combinatorial logic in the memorycontroller 40. Blocks, combinations of blocks, and/or the flowchart as awhole may be pipelined over multiple clock cycles. Updating the agecounters and upgrading the corresponding memory operations may beperformed in parallel for each operation.

The memory controller 40 may assign an age counter to each memoryoperation upon entry of the operation into a queue (e.g. the MCIU queues90 or 92, the PSQ transaction queues 112, and/or the MIF transactionqueues 172) (block 250). The aging counters may be different in eachqueue and need not be carried from queue to queue. Alternatively, anaging counter may be assigned at the MCIU and carried through the otherqueues. The initial value of the aging counter may be based on the QoSparameter of each memory operation (e.g. shorter values for higher QoSvalues). The initial values may be fixed or programmable.

The memory controller 40 may detect an update event that updates a givenaging counter (decision block 252). The update event may vary dependingon how the aging counter is measured. For example, some embodiments maymeasure age in terms of a number of times a memory operation is bypassedby a younger memory operation. In such embodiments, the update event maybe the bypassing of the memory operation. Other embodiments may countclock cycles as age, and the update event may be the rising or fallingedge of the clock. Still other embodiments may define age in other ways,and the update event may be detected accordingly. In response todetecting the update event (decision block 252, “yes” leg), the memorycontroller 40 may decrement the age counter (block 254).

The memory controller 40 may be configured to detect whether or not agiven age counter has expired (decision block 256). If so (decisionblock 256, “yes” leg), the memory controller 40 may be configured toupgrade the QoS level to the next higher level in the same set of QoSlevels (block 258). If the upgraded QoS level is not the highest level(e.g. RTR, decision block 260, “no” leg), the memory controller 40 mayassign a new age counter value based on the new QoS level for furtheraging-based upgrade (block 262). Accordingly, in this embodiment, theQoS level of a memory operation may be upgraded through each interveningQoS level to the highest QoS level as the memory operation ages.

Turning next to FIG. 22, a block diagram of one embodiment of a system350 is shown. In the illustrated embodiment, the system 350 includes atleast one instance of the integrated circuit 10 coupled to externalmemory 12 (e.g. the memory 12A-12B in FIG. 1). The integrated circuit 10is coupled to one or more peripherals 354 and the external memory 12. Apower supply 356 is also provided which supplies the supply voltages tothe integrated circuit 10 as well as one or more supply voltages to thememory 12 and/or the peripherals 354. In some embodiments, more than oneinstance of the integrated circuit 10 may be included (and more than oneexternal memory 12 may be included as well).

The peripherals 354 may include any desired circuitry, depending on thetype of system 350. For example, in one embodiment, the system 350 maybe a mobile device (e.g. personal digital assistant (PDA), smart phone,etc.) and the peripherals 354 may include devices for various types ofwireless communication, such as wifi, Bluetooth, cellular, globalpositioning system, etc. The peripherals 354 may also include additionalstorage, including RAM storage, solid state storage, or disk storage.The peripherals 354 may include user interface devices such as a displayscreen, including touch display screens or multitouch display screens,keyboard or other input devices, microphones, speakers, etc. In otherembodiments, the system 350 may be any type of computing system (e.g.desktop personal computer, laptop, workstation, net top etc.).

Numerous variations and modifications will become apparent to thoseskilled in the art once the above disclosure is fully appreciated. It isintended that the following claims be interpreted to embrace all suchvariations and modifications.

What is claimed is:
 1. A system comprising: a plurality of sources ofmemory operations, where each source is configured to transmit memoryoperations and corresponding quality of service (QoS) parametersindicating a requested QoS for the corresponding memory operations; anda memory controller coupled to the plurality of sources and configuredto control access to a memory for the memory operations, wherein thememory controller comprises a plurality of queues configured to storememory operations, and wherein the memory controller is configured toarbitrate among the memory operations responsive to the QoS parameters,and wherein the memory controller is configured to modify the QoSparameters of pending memory operations in the queues from a firstsource of the plurality of sources responsive to an upgrade QoSindication from the first source, wherein the modified QoS parameter fora given pending memory operation is different from an original QoSparameter provided by the first source with the pending memoryoperation, and wherein the modified QoS parameter indicates a higherlevel of QoS than a level of QoS indicated by the original QoS parameterprovided with the pending memory operation.
 2. The system as recited inclaim 1 wherein the upgrade QoS indication is a new request from thefirst source having a QoS parameter indicating the higher level of QoS.3. The system as recited in claim 1 wherein the upgrade QoS indicationis a sideband interface.
 4. The system as recited in claim 1 wherein theupgrade QoS indication includes an identifier to be compared tocorresponding identifiers that accompanied the pending memoryoperations, wherein the memory operations to experience QoS parametermodification are identified by matching the identifier to thecorresponding identifiers.
 5. The system as recited in claim 4 whereinonly a portion of the identifier is matched to the correspondingidentifiers.
 6. The system as recited in claim 1 wherein the memorycontroller is further configured to assign an age counter to each memoryoperation, and wherein the memory controller is configured modify theoriginal QoS parameter for a given memory operation responsive toexpiration of the age counter.
 7. The system as recited in claim 6wherein an initial value of the age counter for the given memoryoperation is determined responsive to the original QoS parameterassigned to the given memory operation by the first source.
 8. A methodcomprising: receiving a first memory operation in a memory controllerfrom a first source, the first memory operation having a first QoSparameter assigned by the first source indicating a first QoS level forthe first memory operation; determining that the first QoS level is tobe modified for the first memory operation in the memory controllersubsequent to receiving the first memory operation in the memorycontroller; and changing the first QoS parameter to indicate a modifiedQoS level in response to the determining, wherein the modified QoS levelis an upgraded QoS level indicating a higher level of service that thefirst QoS parameter originally indicated.
 9. The method as recited inclaim 8 further comprising receiving a second memory operation from thefirst source in the memory controller, wherein the determining isresponsive to a second QoS parameter corresponding to the second memoryoperation indicating a higher QoS level than the first QoS level. 10.The method as recited in claim 8 further comprising receiving a QoSupgrade request from the first source via an interface that is separatefrom a request interface on which the first memory operation isreceived, wherein the determining is responsive to the QoS upgraderequest.
 11. The method as recited in claim 8 further comprising:assigning an age parameter to the first memory operation responsive toreceiving the first memory operation; detecting that the age parameterhas expired; and wherein the determining is responsive to the detecting.12. The method as recited in claim 11 wherein an initial value of theage parameter is dependent on the first QoS parameter that was assignedby the first source.
 13. The method as recited in claim 12 furthercomprising: assigning a second age parameter to the first memoryoperation responsive to the detecting, wherein an initial value of thesecond age parameter is dependent on the modified QoS level; detectingthat the second age parameter has expired, and changing the first QoSparameter to indicate a second modified QoS level responsive to thesecond age parameter expiring.
 14. A memory controller comprising: aplurality of ports, wherein the memory controller is coupled to receivememory operations on the plurality of ports, each memory operationhaving a quality of service (QoS) parameter indicating a requested QoSlevel for the memory operation; and memory channel circuitry configuredto receive the memory operations from the plurality of ports andconfigured to schedule each memory operation on a memory channel towhich that memory operation is directed, wherein the memory channelcircuitry is configured to select memory operations for scheduling atone or more of a plurality of scheduling points responsive to the QoSparameters of the memory operations, and wherein the memory channelcircuitry is configured to upgrade QoS levels for one or more memoryoperations awaiting scheduling in the memory channel circuitry from anoriginal QoS level assigned by a source of the one or more memoryoperations to a higher QoS level responsive to the higher QoS levelbeing subsequently received for a first port of the plurality of ports,wherein the one or more memory operations were received on the firstport.
 15. The memory controller as recited in claim 14 wherein thehigher QoS level is indicated responsive to receiving another memoryoperation on the first port having a corresponding QoS parameterindicating the higher QoS level.
 16. The memory controller as recited inclaim 14 wherein the first port comprises an interface on which thememory operations are transmitted and a second interface on which thehigher QoS level is requested.
 17. The memory controller as recited inclaim 14 wherein the one or more of the plurality of scheduling pointsinclude a first scheduling point at which the memory operations fromdifferent ports of the plurality of ports are scheduled to the memorychannels, and wherein the first scheduling point is responsive to theQoS parameters for each memory operation.
 18. The memory controller asrecited in claim 17 wherein the one or more of the plurality ofscheduling points include a second scheduling point at which the memoryoperations are scheduled based on affinity, and wherein the secondscheduling point is responsive to the QoS parameters for each readmemory operation but not the QoS parameters for write operations. 19.The memory controller as recited in claim 18 wherein the QoS parametersfor write operations are dropped at the second scheduling point.
 20. Thememory controller as recited in claim 18 wherein the one or more of theplurality of scheduling points include a third scheduling point at whichone or more memory commands are synthesized from each memory operation,and wherein the third scheduling point is not responsive to the QoSparameters for the read memory operations.